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Measurement Properties of Student Social-Emotional Competency and School Culture- 
Climate Surveys in the NewSchools Invent Cohort 


Abstract 


This study describes the measurement properties of two curated surveys, consisting of previously 
developed scales, assessing student perceptions of their social-emotional competencies (SEC) 
and the school culture/climate (CC). The surveys were administered through the Transforming 
Education/NewSchools Venture Fund partnership that worked with the NewSchools Invent 
Cohort, a network of 23 schools across 11 states, to expand the definition of student success. The 
analytic sample for this study includes more than 3,000 students in grades 4 through 12 at 18 
schools surveyed in the 2016-17 school year. Using classical test theory as our conceptual and 
analytic lens, we conducted tests of internal consistency, item discrimination, omit rates, 
differential item functioning, and exploratory and confirmatory factor analysis. Our findings 
provide initial evidence of the validity of the survey scores for measuring student perceptions of 
their SEC and the school CC in the NewSchools Invent schools, suggesting that the surveys are 
suitable for school and cohort-level decision-making related to students’ social emotional 
competencies and experiences within the school. 
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Introduction 

Research has demonstrated the importance of intra- and interpersonal social-emotional 
competencies (SEC) for positive academic and life outcomes. Skills such as self-regulation and 
social competence and motivational orientations such as growth mindset predict outcomes 
ranging from high school and college completion to employability and earnings, as well as 
physical and mental health (see reviews in Gabrieli, Ansel, & Krachman, 2015; Jones, 
Greenberg, & Crowley, 2015; Moffit, et al., 2011). We also know that these skills and mindsets 
do not develop in a vacuum. Aspects of a school’s culture/climate (CC)—for example, students’ 
perception of sense of belonging and teacher-student relationships—can foster students’ 
academic and SEC development (Anderman, 2003; Berg et al., 2017; Konishi et al., 2010). 
Students with strong SEC are more likely to build positive relationships with peers and adults, 
participate in classroom activities, and engage in learning (Jones & Kahn, 2017). In turn, schools 
that are intentional about combining rigorous academics with social-emotional development 
produce deeper, longer-term student learning (Farrington et al., 2012; Jones & Kahn, 2017). 

To rigorously examine the interconnections between academic outcomes, social- 
emotional competency development and school environment, the field needs valid and reliable 
measures of student SEC and the school CC across grade levels and demographic subgroups. In 
partnership with NewSchools Venture Fund, Transforming Education administered a set of pre- 
existing self-report SEC scales and school CC scales as two new comprehensive student 


surveys.! In this study, we examine the measurement properties of the SEC and CC surveys, 


' Transforming Education (TransformEd) and NewSchools Venture Fund (NewSchools) have partnered together in 
an effort to help schools in the NewSchools Invent cohort address a broader range of factors that contribute to 
students’ long-term success. This three-year partnership is focused on achieving four key goals: (1) Provide 
actionable data and research to help school leaders expand the definition of student success and to improve 
outcomes for students on a range of indicators that relate to long-term success; (2) Provide support (e.g., resources, 
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based on their administration in and use with the NewSchools Invent cohort. While there exists 
prior evidence of the reliability and validity for each individual SEC and CC scale, the validity of 
the new surveys has not been examined. We recognize that validation is an ongoing process of 
accumulating multiple sources of evidence to support the appropriateness of the decisions and 
inferences being made with the instrument (Kane, 2006; 2013; Messick, 1989). In this study, we 
examine the measurement properties of the surveys to provide evidence on a key aspect of 
validation — how well the measures were designed; that is, the extent to which the items provide 
consistent and new information about the underlying constructs being assessed, the extent to 
which the items are interpreted comparably across student subgroups, and the extent to which 
scales for each survey are measuring unique constructs. To conduct this study, we partnered with 
Education Analytics, an organization comprised of leading measurement experts who serve as 
the analytic partner to the CORE Districts. The set of analyses we include in this report were 
informed by analyses that EdAnalytics conducted on the CORE Districts’ SEC survey (see 
Meyer, Wang & Rice, 2018).? 
Background 

There are different methods of assessing student SEC and school CC; student self-reports 
are one method that can provide insight into student perceptions of skills, mindsets, and 
experiences. Social and cognitive psychology literature suggests that self-reported questionnaires 


provide a good medium for respondents to communicate their true perceptions (Krosnick, 1999). 


connections, ideas, etc.) to help school leaders change practices based on research and data; (3) Provide data to 
NewSchools to help them understand their portfolio on a variety of metrics so they can identify trends and inform 
their board, investment partners, and funders; and (4) Contribute to the broader national dialogue in the field about 
how to expand the definition of student success in research, policy and practice. 

2 The CORE districts are a collaborative of eight California school districts that have jointly developed a system of 
school accountability and continuous improvement that includes measures of social-emotional skills based on 
student self-reports. 
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In particular, self-report questionnaires “are arguably better suited than any other measure for 


assessing internal psychological states like feelings of belonging” (Duckworth & Yeager, 2015, 
p. 5). Furthermore, numerous studies have shown that results from self-report questionnaires are 
predictive of objectively-measured outcomes (Fredricks et al., 2011; Hanson, & Kim, 2007; 
Squires et al., 2011; West, Buckley, Krachman, & Bookman, 2017). On a practical level, self- 
report surveys are an easy and low-cost way of obtaining information about individual 
perceptions, attitudes, and behavior. 

While student self-reports provide a practical and scalable solution for assessing student 
social-emotional competencies and illuminating student perceptions, researchers have raised 
questions about potential biases inherent in the self-reports (Duckworth & Yeager, 2015). West 
et al. (2017) recommended taking these concerns seriously but also emphasized the fact that self- 
reports provide a tremendous learning opportunity for the field. Further, a recent report by Policy 
Analysis for California Education (PACE) at Stanford University found promising evidence for 
the validity of student self-reports of social-emotional competencies administered in the CORE 
districts as part of CORE’s continuous improvement system (Gehlbach & Hough, 2018). 

Addressing concerns about self-report questionnaires requires continuous testing and 
validation of existing measures to ensure their proper use. In order to better serve students and to 
help schools make effective data-informed decisions, by, for example, linking formative SEC 
assessments to specific instructional practices, it is imperative that we use measures of student 
SEC and school CC that are able to assess such constructs equally well across age, gender, 
race/ethnicity, and socio-economic groups (Phillips & Rowley, 2015). Furthermore, validation of 
SEC and CC measures is an important prerequisite to examining theoretical relationships among 


student academic performance, student SEC, and school CC. In this study, we examine the 
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measurement properties of the SEC and CC survey items and scales using classical test theory. In 
line with Meyers et al., (2018), we provide a thorough review of properties of the items and 
scales to investigate the extent to which surveys can be used reliably for school- and cohort-level 
decision-making about how to select and implement strategies, resources, and policies that 
improve student outcomes. 
Data 

Instrumentation 

In the summer of 2016, TransformEd partnered with school leaders who were members 
of the NewSchools Invent cohort of innovative schools. The goal of this partnership was to help 
school leaders expand their definition of student success through the collection and analysis of 
data on student social-emotional competencies and school culture-climate factors. The first step 
toward achieving this goal was to determine which SECs and CC factors should be assessed 
across the cohort of schools based on common measures. From interviews with school leaders in 
the NewSchools Invent cohort, TransformEd generated a list of more than 60 constructs that 
school leaders believed to be most important to students’ long-term success. TransformEd then 
filtered this list through its “3Ms framework,” narrowing the list to constructs that are 
meaningful (have an impact on long-term student outcomes), measurable (can be assessed in a 
school setting), and malleable (can be developed in a school setting), based on existing literature 
from the fields of economics, psychology, human development, and education.’ The final set of 
constructs include seven social-emotional competencies (e.g., self-regulation and growth 


mindset) and seven school culture-climate factors (e.g., school safety and student-teacher 


3 See https://www.transformingeducation.org/wp-content/uploads/2017/06/Tool-3.pdf 
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relationships). 

TransformEd scanned the field to identify a set of pre-existing scales (1.e., validated and 
free to use for non-commercial research purposes) that measured student perceptions of these 14 
prioritized constructs. The SEC scales were drawn from leading researchers and are used in 
surveys administered by the CORE Districts, the Boston Charter Research Collaborative 
(BCRC), Washoe County School District, and the National Assessment of Education Progress 
(NAEP).’ The school CC scales were drawn from Panorama Education’s school climate survey 
and the U.S. Department of Education’s ED School Climate Survey (EDSCLS).° TransformEd 
assembled the scales into a new SEC survey and CC survey to administer to students in schools 
that were part of the NewSchools Invent cohort in the 2016-17 school year. See Appendix A for 
the full set of scales, items, and sources. 

Student-reported SEC survey. A 39-item SEC survey was used to measure student 
perceptions of seven different competencies.° For each item, students selected one of five Likert- 
type options (strongly disagree to strongly agree). 

- Curiosity (6 items, e.g., “I like activities that challenge my thinking abilities”)’ 


- Growth mindset (4 items, e.g., “Challenging myself won’t make me any smarter”)® 


4 For further background on our partnership with the CORE Districts and BCRC, please see: 
https://www.transformingeducation.org/our-work/with-schools-and-systems/core-districts-partnerships/ and 
https://www.transformingeducation.org/our-work/with-schools-and-systems/bere-partnership/. For more information 
on Washoe County School District’s SEL work, please see: http:/Awww.wcesddata.net/data-topics/sel/. For further 
information on NAEP’s SEC scales, please see https://nces.ed.gov/nationsreportcard/tdw/instruments/noncog.aspx 
5 For further background on the school climate scales, please visit: https://www.panoramaed.com/school-climate- 
survey and https://nces.ed.gov/surveys/edscls/pdf/EDSCLS Student _Questionnaire English.pdf. 

® The full list of SEC items for each scale are listed in Table 5. See Tables A2 and A3 in the appendix for overall 
and subgroup-level means and standard deviations of each SEC and CC competency. 

7 The curiosity scale was drawn from the National Assessment of Educational Progress (NAEP). NAEP refers to this 
construct as “desire for learning.” 

8 The growth mindset scale was developed by Camille Farrington of the University of Chicago Consortium on 
Chicago School Research (CCSR), building upon the work of Carol Dweck of Stanford University. Items in this 
scale are reverse-coded. 
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- Perseverance (4 items, e.g., “I finish whatever I begin”)? 
- Self-awareness (4 items, e.g., “Please let us know how easy or difficult each of the 
following are for you: “Knowing what my strengths are”)!° 
- Self-efficacy (4 items, e.g., “I can earn an A in my classes”)! 
- Self-regulation (9 items, e.g., “I came to class prepared”)'* 
- Social awareness (8 items, e.g., “I listened carefully to other people’s points of view”)! 
Student-reported culture-climate survey. The 36-item culture-climate survey was used 
to measure student perceptions of the school CC across seven domains. For each item, students 
select one of five Likert-type options, (strongly disagree to strongly agree).'4 
- Cultural and linguistic competence (5 items, e.g., “Boys and girls are treated equally 


well”)!5 


° The perseverance scale was developed by Angela Duckworth of the University of Pennsylvania. Duckworth refers 
to this construct as “grit.” The original scale consisted of the items included in the SEC survey, however it has since 
been modified to include two factors: consistency of interest and perseverance of effort. Our scale aligns closely 
with the items that fall under perseverance of effort factor (see https://www.dropbox.com/s/mS5wo3y0iisOqtf/8- 
item%20Grit%20Scale_ Child%20Adapted%20Version 4.pdf?dl=0). 

'0 The self-awareness scale was developed by Washoe County, American Institutes for Research (AIR) and the 
Collaborative for Academic, Social, and Emotional Learning (CASEL) as part of the Collaborative Districts 
Initiative (CDI. 

11 The self-efficacy scale was developed by Camille Farrington of the University of Chicago Consortium on School 
Research (CCSR). 

'2 The self-regulation scale was developed by Angela Duckworth of the University of Pennsylvania and adapted by 
Clancy Blair of New York University (Duckworth, 2008; Blair & Diamond, 2008). Duckworth and Blair refer to 
this construct as “self-control” (Duckworth, Gendler, & Gross, 2016). While there are substantive differences in the 
underlying constructs of self-control and self-regulation (see for example, 
https://www.psychologytoday.com/us/blog/self-reg/20 1607/self-reg-self-regulation-vs-self-control), we refer to the 
scale as self-regulation based on feedback from the school leaders taking part in this study. 

'3 The social awareness scale was developed by CASEL and AIR and adapted by Hunter Gehlbach for use by the 
CORE Districts. 

'4 Three of the SEC and CC scales are reverse coded: the growth mindset scale (due to negatively framed questions), 
the cultural and linguistic competence scale (with response options listed as strongly agree to strongly disagree and 
school safety scale (with questions asking about lack of school safety within a school). 

15 The cultural and linguistic competence scale was drawn from the EDSCLS. It is based on a 4-point Likert Scale. 
For the purposes of our survey, we converted it to a 5-point Likert scale to be consistent with the other six CC scales 
drawn from Panorama. 
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- Learning strategies (5 items, e.g., “Before you start on a challenging project, how often 
do you think about the best way to approach the project?’’) 
- Rigorous expectations (5 items, e.g., “How often do your teachers make you explain your 
answers?”)!¢ 
- School safety (6 items, e.g., “How often do you worry about violence at your school?”) 
- Sense of belonging (5 items, e.g., “How connected do you feel to the adults at your 
school?”’) 
- Student engagement (5 items, e.g., “How excited are you about going to your classes?’’) 
- Teacher-student relationships (5 items, e.g., “How many of your teachers are respectful 
towards you?’’) 
Administration 
The surveys were administered through an online platform, managed by Panorama 
Education, to students in grades 4-12 across 18 schools in the NewSchools Invent cohort in 
2016-17. See Appendix B for summary statistics for each of the 14 competencies from the 
Spring 2017 survey administration. Students took the survey in the first 4-6 weeks of the fall 
semester school and again in the spring semester during the last 4-6 weeks of school. Students 
were asked to complete both surveys in one sitting, which generally took approximately 45 
minutes. While there was no official protocol on where and when students took the surveys, most 
schools chose to administer surveys during an advisory or homeroom period to minimize the 


impact on academic instructional time. 


'6 The learning strategies scale, rigorous expectations scale, school safety scale, sense of belonging scale, student 
engagement scale, and teacher-student relationship scale were developed by Panorama Education (see 
(https://www.panoramaed.com/school-climate-survey). 
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Analytic Sample 


Table 1 summarizes the demographic characteristics of our analytic sample based on 
student roster results. Note that our analytic sample excludes students from one eligible school 
that chose not to administer the surveys, as well as students in grades 4-12 who were eligible to 
take the surveys but chose not to or were absent on the days the surveys were administered. We 
also excluded students who took the survey in Spanish due to small sample sizes.'’ Our final 


analytic samples include all students who answered at least one item on each survey in each time 


period. 
Table 1. Demographic characteristics of students in the analytic sample 
Characteristic Fall 2016 Sample Spring 2017 Sample 
with Student Survey | with Student Survey 
Data Data 
Student | Student | Student | Student 
SEC CC SEC CC 
Number of schools 18 18 18 18 
Number of students (Grades 4 through 12) n=3,222 n=3,274 n=3,021 n=3,086 
Female 48% 48% 49% 50% 
Latino 53% 52% 53% 52% 
Black or African American 20% 20% 18% 18% 
Asian 3% 3% 3% 3% 
Two or More Races 1% 1% 1% 1% 
White 19% 20% 21% 22% 
Other 0.53% 0.52% 0.53% 0.49% 
Middle Eastern North A frican(MENA) 3% 3% 3% 3% 
ELL 17% 18% 18% 18% 
FRPL 65% 65% 64% 64% 
SPED 8% 8% 8% 9% 


'7 For the Fall analytic sample, we exclude sixteen students (0.49%) who took the SEC survey in Spanish, and 
eighteen (0.55%) who took the CC survey in Spanish. For the Spring analytic sample, we excluded seven students 
(0.23%) who took the SEC survey in Spanish and six students (0.19%) who took the CC survey in Spanish. 
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Methods 

We used several diagnostic methods in line with classical test theory to examine the 
measurement properties of the SEC and CC survey items and scales (Hambleton & Jones, 1993; 
Jiang, Wang, & Weiss, 2016). We conducted most of our psychometric analyses based on 
classical test theory instead of item response theory (IRT) due to sample size limitations at the 
subgroup level (student demographics and grade level). Studies suggest that a sample size of 400 
or greater is required to obtain estimates with useful standard errors (He & Wheadon, 2013; 
Jiang, Wang, & Weiss, 2016). 

To assess the quality of SEC and CC items and scales, we first examined the internal 
consistency of each SEC and CC scale by conducting tests of reliability using Cronbach’s alpha 
coefficient of each scale. We then conducted three psychometric diagnostics using the statistical 
software R and Stata. First, we examined item response spread to identify the proportion of 
students that endorsed a particular item. The purpose of this analysis was to determine whether 
the items included in our scales are able to capture the full range of student perceptions. Second, 
we examined item discrimination to determine whether items adequately differentiate between 
students with high perceptions (of SEC or CC) and low perceptions (of SEC or CC). This type of 
analysis can illuminate the extent to which the items are cohesively measuring the underlying 
construct. Third, we analyzed omit rates to identify items that were systematically skipped by 
students. Analyzing omit rates is important because if items within a scale are systematically 
skipped by a considerable number of students, it could bias aggregate responses of the scale 
(Phillips, Reddy, & Durning, 2016). 

In addition, we conducted differential item functioning (DIF) analysis on the survey 


items to identify whether there are statistically significant differences in student performance on 
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an item between the reference group (e.g., male) and the focal group (e.g., female) among 


students with similar levels of self-perceived SEC competencies or similar perceptions of school 
CC (Meyers et al., 2018). 

DIF is an important analytic tool because it provides insight into the extent to which the 
survey is capable of producing comparable responses across student subgroups (Guerra & Jagers, 
1998; Rogers, 2014). This analysis is particularly relevant for the SEC and CC surveys as its 
participants are a diverse group of students based on their race/ethnicity, gender, and grade level. 
As such, it is possible that students from a particular subgroup may answer an SEC or CC item in 
a particular way not because of their true perception, but because of societal factors that 
influence their beliefs in how they should respond (i.e., social desirability bias) or because of 
difficulty in understanding a particular word or phrase. Jeong & Lee (2016) provided an example 
of differences in Likert-type responses on a self-confidence item between respondents from a 
country where humility is encouraged versus a country where self-assurance is encouraged. They 
used DIF to identify such unequal responding patterns among groups. While DIF does not 
necessarily indicate bias, it may simply highlight important differences across subgroups in 
students’ perceptions or experiences of a particular aspect of the school culture-climate. 
Identifying and further investigating such items is an important step toward creating a survey that 
produces comparable scores across all student subgroups (Raju et al., 2002). 

Lastly, we performed exploratory and confirmatory factor analyses to assess whether the 
scales within each survey were measuring distinct constructs. Exploratory factor analysis (EFA) 
assumes no underlying factor structure and therefore allows us to freely estimate the factor 
structure across all items within a survey. Confirmatory factor analysis (CFA) allows us to 


impose theoretically expected factor structures to test the hypothesis that the desired relationship 


I2 
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between observed survey items and their underlying latent constructs exists. With the SEC and 
CC surveys, we impose a 7-factor structure since each includes seven distinct scales. If, through 
EFA and CFA, two scales are found to be measuring the same underlying construct, it suggests 
that the two scales can be combined or that one of the scales can be removed. 

Findings 
Internal consistency 

We assess internal consistency by examining Cronbach’s alpha coefficient for each of the 
survey scales. Cronbach’s alpha coefficient (a) assesses the extent to which items within a scale 
measure the same construct. In general, a higher Cronbach’s alpha coefficient indicates greater 
consistency of item responses.'* Alpha coefficients of 0.60 to 0.70 are considered common in 
survey research (Lamb et al., 2012; Ravens-Sieberer et al., 2008). 

Generally, we find that the scales in the SEC and CC surveys demonstrate a high degree 
of internal consistency across all students in the NewSchools Invent cohort. For SEC scales, 
alpha coefficients range from 0.70 to 0.92 for grades 4 through 12. For CC scales, alpha 
coefficients range from 0.66 to 0.91 for grades 4 through 12. See Appendix C for grade-level 
alpha coefficients for both SEC and CC surveys. 

Item response spread 

The item response spread statistic indicates the degree to which students endorse an item. 
It is calculated as the proportion of the maximum obtainable score and ranges between 0 and 1. If 
an item is endorsed by a large proportion of students (1.e., most students select a 4 or a 5 ona 5- 


point Likert scale) or an item is endorsed by a small proportion of students (i.e., most students 


'8 Cronbach’s alpha coefficient provides an imperfect measure of internal reliability, since it is affected by the 
number of items included in the scale, and high or perfect correlations can suggest item redundancy and/or a narrow 
measure of the underlying construct (see Tavakol & Rennick, 2011). 
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select a 1 or a 2 ona 5-point Likert scale), this can be an indicator that the item is either too 
“easy” or too “difficult” to provide meaningful information about the relevant construct being 
measured. For the purposes of this study, we use the threshold of 0.30 to 0.80 for a constructed 
response item (see Meyer et al., 2018), whereby an item is deemed too “difficult” if the average 
endorsement is below 30 percent and too “easy” if average endorsement is above 80 percent. 

While the terms “hard” and “easy” are generally used for cognitive items with correct or 
incorrect answers, the meaning can be less intuitive for items assessing SEC and CC constructs, 
particularly those assessing non-skill factors. That said, in line with psychometric work on SEL 
surveys conducted by Meyer et al. (2018) and Davidson et al. (2016), we use these terms since 
they are accepted terms for this type of analysis.'? In the case of the SEC and CC items, these 
terms are used to indicate whether most respondents indicated less favorable or more favorable 
perceptions of their SEC or the school CC based on Likert-type response options. The item 
difficulty metric provides useful information in that it indicates whether the item contributes 
meaningful information about the relevant construct being measured. 

Table 2 provides the list of items that are identified for further analysis based on the high 
or low proportion of students in our analytic sample that endorsed each survey item. Looking at 
endorsement rates by grade level, we do not find any item with endorsements below the 
threshold of 0.30, but we find 14 items with endorsements above the common threshold of 0.80, 
indicating that most students have favorable perceptions of the factor being assessed. Table 2 
also shows that the majority of items with above 80 percent endorsement belong to one SEC 


scale (1.e., self-regulation) and two CC scales (i.e., rigorous expectations and school safety). 


'9 See Washoe County School District’s Social Emotional Learning webpage for a practical application of this type 
of analysis: http://www.wcsddata.net/data-topics/sel/ 
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While we flag the items below for further diagnostics, we consider multiple aspects of item 


characteristics to decide whether any of these items are potentially problematic to include in 
future surveys. 


Table 2. Item Response Spread: Items identified for further diagnostics 


Construct Survey Item Wording Spread Grades 
Range Flagged 
Growth Mindset SEC If I am not naturally smart in a subject, I will never do 0.84 4 
well in it.* 
Self-Regulation SEC I came to class prepared. 0.81-0.84 | 5,11, 12 
Self-Regulation SEC I remembered and followed directions. 0.83 -0.85 | 11, 12 
Self-Regulation SEC I allowed others to speak without interruption. 0.82 12 
Self-Regulation SEC I was polite to adults and peers. 0.81-0.87 | 4,5, 6, 11, 12 
Social Awareness SEC I cared about other people's feelings. 0.82 4 
Rigorous Expectations | CC When you feel like giving up on a difficult task, how 0.83 4 
likely is it that your teachers will make you keep trying? 
Rigorous Expectations | CC How much do your teachers encourage you to do your 0.80 - 0.84 | 4,5 
best? 
Rigorous Expectations | CC How often do your teachers take time to make sure you | 0.81 4 
understand the material? 
School Safety CC How often do students get into physical fights at your 0.82 - 0.87 | 9, 10, 12 
school?* 
School Safety CC How likely is it that someone from your school will 0.81-0.90 | 4,5, 8-12 
bully you online?* 
School Safety CC How often do you worry about violence at your 0.81 - 0.87 | 9, 10, 11, 12 
school?* 
School Safety CC If a student is bullied in school, how difficult is it for 0.82 12 
him/her to get help from an adult?* 
Teacher-Student CC How many of your teachers are respectful towards you? | 0.80-0.83 | 4,5, 11 
Relationship 


*Reverse-coded item 


Item discrimination 

We examine item discrimination using item-total correlation based on classical test 
theory. Item-total correlation is the correlation between a given item and all the items in the 
scale. It is expected that a student who responds to an item with a 4 or 5 on a Likert scale should 
also have, in general, higher overall scores on the construct compared to a student who responds 
with a 2 or 3 on that item. This metric indicates how well an item is able to discriminate between 


students with low and high perceptions of the construct being measured. 
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greater than the common threshold of 0.40 for survey responses (Ladhari, 2010). On the SEC 


We find that all items have adequate discrimination, with item-total correlations much 


survey, item-total correlations range from 0.64 to 1 among students in grades 4-12. For the CC 
survey, item-total correlations range from 0.52 to 0.97 for students in grades 4-12. 
Omit rate 

Examining omit rates for each survey item at each grade level allows us to identify items 
that are skipped by a large proportion of students. In cases where students are systematically 
omitting an item or entire scale, it can bias our interpretation of the aggregate response of a scale. 
We find high response rates at the item level across both SEC and CC surveys, with at least 96 
percent of students responding to all items in the survey. For the four percent of data that is 
missing, we do not find any consistent patterns in which items are skipped by grade level. We 
find that students in grade 4 and grade 6 tend to skip more items throughout the survey than any 
other grade. In grade 4, the self-regulation item “I remained calm even when criticized or 
otherwise provoked” (Q6) is the most commonly skipped item. In grade 6, the curiosity items “I 
like activities that challenge my thinking abilities” (Q2) and “TI find satisfaction in thinking hard 
and for long hours” (Q4) are the most commonly skipped items. Other items with relatively high 
skip rates among sixth graders include “Overall, how well do your learning strategies help you 
learn more effectively?” (learning strategies Q4); “My intelligence is something that I can’t 
change very much” (growth mindset Q3); and “I am diligent. I never give up” (perseverance 


Q4). Nonetheless, responses rates for these item/grade combinations are still quite high, ranging 
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from 96 to 100 percent.”° One hypothesis is that the items skipped most frequently are those that 
contain words or concepts that may be difficult for students to understand. 
Differential Item Functioning (DIF) 

We conduct DIF analysis using the standardized mean difference (SMD) method in 
conjunction with the Mantel chi-square statistic (Dorans & Kulick, 1986; Dorans & Schmitt, 
1991; Mantel, 1963; Mantel & Haenszel, 1959). The SMD method was originally developed by 
Dorans and Kulick (1986) to estimate DIF for dichotomous items. Dorans and Schmitt (1991) 
extended this statistic to estimate DIF for items with more than two possible scores (e.g., Likert- 
type response choices like those used in our surveys). An item is classified as having moderate- 
to-large DIF if the Mantel chi-square p-value is below 0.05 and the ratio of absolute value of 
SMD and the standard deviation of the item is higher than 0.25 (Dorans & Schmitt, 1991; Zwick 
& Ercikan, 1989). To ensure a large enough n-size, we combine adjacent grade levels: 4-5 
(elementary), 6-8 (middle), 9-10 (early high school), and 11-12 (late high school), and calculate 
DIF statistics only for those where both the focal group and the reference group have at least 100 
students who completed all items within the scale. Table 3 lists the focal and standard reference 


comparison groups for DIF.7! 


20 We do not look at omit rates for demographic subgroups because of high response rates overall and at the grade 
level. 

>! The reference group is typically chosen based on the types of differential item functioning one is trying to 
disprove (Holland and Wainer, 1993). Historically, the normativistic models based on White, middle-class children 
have been dominant in the fields of developmental psychology and child development which can lead to survey 
development which privileges the experience of White male students above others. Following from this, our primary 
concern is that the items were designed based on a White- and male-dominant conception of strong SEC and 
favorable school CC. To test whether the survey is biased toward these groups, we include White students as our 
reference group when testing DIF by race/ethnicity, and we include males as our reference group when testing DIF 
by gender. 
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Table 3. DIF Comparison Groups 


Grouping Variable Focal Group Reference Group 
Gender Female Male 
Ethnicity Black or African American White 
Latino White 


Socioeconomic Status 
Special Instructional Needs 


Administration Term 


Students who receive free/reduced- 


price lunch 


English language learners (ELL) 


Students with disabilities 
Fall 


Students who do not receive 
free/reduced-price lunch 
Non-ELL 

Students without disabilities 
Spring 


An item with moderate-to-high DIF is often considered a concern when it is measuring 
different abilities for the focal group after controlling for overall differences in survey scores 
(Dorans & Schmitt, 1991). Table 4 provides a summary of the DIF items and the respective focal 
subgroups where we find moderate-to-high DIF from survey administrations in the Fall of 2016 
and/or Spring of 2017. Of the 39 SEC items, we find four items with moderate-to-high DIF for 
one or more focal group(s) based on the threshold specified above. For example, one self- 
regulation item, “I kept my temper in check,” was identified in both the Fall and Spring but for 
different focal groups and at different grade levels. Similarly, one social awareness item, “I cared 
about other people’s feelings,” was identified as having moderate-to-high DIF in Spring 2017 for 
Black/African American students in middle school. 

Among the 36 culture-climate items, five of the six school safety items were identified in 
different DIF analyses across elementary and middle school grade levels and among different 
subgroup comparisons. Of those five school safety items, the item “How often do you worry 
about violence at your school?” was identified in three different demographic subgroup analyses 
with moderate-to-high DIF for ELL, FRPL, and Latino students. Similarly, the item “If a student 
is bullied in school, how difficult is it for him/her to get help from an adult?” in the school safety 


scale had moderate-to-high DIF for ELL students in elementary school, middle school, and early 
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high school in the Fall. It also had moderate-to-high DIF for ELL students in middle school in 


the Spring and for girls in early high school in the Spring. 


Table 4. Items with moderate-to-high DIF that are differentially difficult for focal group(s) 


Survey Construct Item Moderate-to- Grade Term 
high DIF for Level 
Subgroup 
SEC Curiosity I like complex problems more than easy problems. | FRPL Elementary | Fall 2016 
Curiosity I enjoy situations where I will have to think about SPED Early HS Fall 2016 
something. 
Growth My intelligence is something that I can't change FRPL; Black or | Middle Fall 2016 
Mindset very much. African School 
American 
Self-Awareness | Knowing the emotions I feel. Female Late HS Spring 2017 
Self-Efficacy I can earn an A in my classes. FRPL Early HS Spring 2017 
Self-Regulation | I was polite to adults and peers. Black or African | Elementary | Fall 2016 
American 
Self-Regulation | I kept my temper in check. Female; Black or | Middle; Both 
African Late HS 
American 
Social I cared about other people's feelings. Black or African | Middle Spring 2017 
Awareness American School 
Social I was able to describe my thoughts and feelings in | Female Early HS Fall 2016 
Awareness ways that others understood. 
Social I respected the views of others even if I disagreed Black or African | Elementary | Fall 2016 
Awareness with them. American 
CC Engagement In your classes, how eager are you to participate? Latino Middle Fall 2016 
School 
Rigorous How often do your teachers make you explain your | ELL Middle; Fall 2016 
Expectations answers? Early HS 
School Safety How often do students get into physical fights at Latino Middle Spring 2017 
your school? School 
School Safety How likely is it that someone from your school Female Late HS Fall 2016 
will bully you online? 
School Safety How often do you worry about violence at your ELL; FRPL; Elementary; | Both 
school? Latino; Black or | Middle 
African 
American 
School Safety If a student is bullied in school, how difficult is it ELL; Female Elementary; | Both 
for him/her to get help from an adult? Middle; 
Early HS 
School Safety At your school, how unfairly do the adults treat the | Black or African | Elementary | Fall 2016 
students? American 
Teacher- If you came back to visit class three years from Latino Middle Spring 2017 
Student now, how many of your teachers would be excited School 
Relationship to see you? 
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Factor Analysis 


We use exploratory factor analysis (EFA) to examine the dimensionality of the full 
surveys measuring SEC and CC. EFA allows us to uncover the internal structure of a survey by 
indicating how groups of items hang together within the full instrument. We then proceed with 
confirmatory factor analysis (CFA) to validate the hypothesized model structure and examine 
“goodness of fit” indices, including the comparative fit index (CFI), Tucker-Lewis Index (TLI), 
root-mean square error of approximation (RMSEA), and standardized root mean square residual 
(SRMR). 

The first step in factor analysis is to determine whether the data is suitable for factor 
analysis. We use the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (Kaiser, 1974) 
and Bartlett’s test of sphericity to assess the suitability of data. The KMO is a statistic that allows 
us to measure the proportion of variance among our survey items that might be indicative of 
underlying latent common constructs. The KMO test returns values between 0 and 1, with values 
greater than 0.80 indicating adequate sampling. Bartlett’s test approximates a chi-square 
distribution and tests the hypothesis that the variables being analyzed are independent. Data is 
deemed appropriate for factor analysis if the p-value is less than 0.05. We find that our survey 
data satisfies both KMO (0.96) and Bartlett’s test (p < 0.05) criteria, suggesting that our sample 
is adequate and appropriate for factor analysis. 

Exploratory factor analysis. After testing for sampling adequacy and data suitability, 
we conduct EFA to empirically examine the structure of the survey questions. EFA allows us to 
explore the factor structure of SEC and CC constructs that emerge from the data without a priori 
specifying any theories about the constructs. It is a common rule-of-thumb in EFA to use factor 


loadings of 0.30 or greater to identify practically significant factor loading (Schmitt & Sass, 
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2011). We use this criteria to identify factor structures and loadings of items to factors for both 


SEC and CC. 

Table 5 presents the factor structure and loadings of items to factors for the SEC survey. 
As it illustrates, our EFA examination results in seven factors. Four of the seven factors clearly 
group into distinct scales forming the growth mindset, self-awareness, self-efficacy, and social 
awareness scales. However, three of the factors—self-regulation, perseverance, and curiosity— 
have overlapping items loading onto two factors. Specifically, three perseverance items (Q1, Q2, 
Q3) load onto self-regulation and two perseverance items (Q3, Q4) load onto curiosity. 

Table 6 presents the factor structure and loadings of items to factors for the CC survey. 
As seen in Table 6, three of the seven factors clearly group into three scales, forming the learning 
strategies, engagement, and rigorous expectations scales. However, four of the factors, namely 
cultural and linguistic competence, teacher-student relationship, school safety, and sense of 
belonging, have overlapping items loading onto two factors. Specifically, one school safety item 
loads slightly more strongly (0.32) onto cultural and linguistic competence compared to school 
safety (0.29) and one sense of belonging item loads more strongly (0.38) onto teacher-student 
relationship compared to sense of belonging (0.23). 

Confirmatory factor analysis. We use CFA to test the a priori hypothesized seven- 
factor model based on seven SEC and seven CC scales that were curated from the prior surveys. 
Using Spring 2017 survey data, we fit two models: one for the SEC constructs and the other for 
the CC constructs. We use an R package called /avaan to run CFA models at each grade level 
separately. We treat the data as ordinal, use a three-stage weighted least squares (WLS) 


estimation approach, and set the constructs to be correlated (Forero et al., 2009). 
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comparative fit index (CFI) to control for the effects of sample size; (2) the root mean square 


We use the following criteria to evaluate model fit (Holgado-Tello et al., 2010): (1) the 


error of approximation (RMSEA) to examine how well the model fits the populations covariance 
matrix; (3) the Tucker-Lewis Index to assess the correspondence between the proposed model 
and the data; and (4) the standardized root mean square residual (SRMR). Acceptable threshold 
for these indices are as follows: 0.95 or larger for CFI and TLI; 0.06 or smaller for RMSEA; and 
0.08 or smaller for SRMR (Hu & Bentler, 1999; Schreiber et al., 2006). 

Table 7 provides details about fit indices for both surveys at each grade level. Results of 
CFA indicate that the hypothesized model appears to be a good fit to the data. Grade-level 
analyses of the four fit indices for grades 4 to 12 confirm good fit, with overall indices 
performing within acceptable thresholds. For the SEC and CC models, the CFI indices range 
from 0.98 to 1.00; TLI ranges from 0.98 to 1; RMSEA ranges from 0 to 0.06; and SRMR ranges 
from 0.05 to 0.08. Our examination of modification index shows that all SEC items have 
modification indices below 100, which is the acceptable threshold (Hooper et al., 2008). At the 
grade level, a few items have modification indices above 100 in some grade levels, but there is 


no consistent pattern across grades. 
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Table 5. Exploratory Factor Analysis of SEC Student Survey: Factor Loadings with Oblimin 


Rotation 


Item Wording 


Factor Loadings 


3 


4 


5 


Self-Regulation 

I came to class prepared. 

I remembered and followed directions. 

I got my work done right away instead of waiting until the last minute. 
I paid attention and resisted distractions. 

I worked independently with focus. 

I remained calm even when criticized or otherwise provoked. 

I allowed others to speak without interruption. 

I was polite to adults and peers. 

I kept my temper in check. 


0.36 


0.47 


Self-Efficacy 

I can earn an A in my classes. 

I can do well on all my tests, even when they are difficult. 
I can master the hardest topics in my classes. 

I can meet all the learning goals my teachers set. 


0.75 
0.90 
0.85 
0.68 


Perseverance 

I finish whatever I begin. 

I work very hard. I keep working when others stop to take a break. 

I stay interested in my goals, even if they take a long time (months or years) to 
complete. 

I am diligent. I never give up. 


0.45 
0.45 


0.32 
0.27 


0.11 
0.28 


0.39 
0.38 


Growth Mindset 

Challenging myself won't make me any smarter. 

There are some things I am not capable of learning. 

My intelligence is something that I can't change very much. 

If I am not naturally smart in a subject, I will never do well in it. 


0.73 
0.78 
0.78 
0.75 


Curiosity 

I like complex problems more than easy problems. 

I like activities that challenge my thinking abilities. 

I enjoy situations where I will have to think about something. 
I find satisfaction in thinking hard and for long hours. 

I enjoy thinking about new solutions to problems. 

I like to think of my life as a puzzle that I must solve. 


0.71 
0.77 
0.78 
0.81 
0.75 
0.61 


Self Awareness 

Knowing what my strengths are. 

Knowing ways I calm myself down. 

Knowing when my feelings are making it hard for me to focus. 
Knowing the emotions I feel. 


Social Awareness 

I listened carefully to other people's points of view. 

I cared about other people's feelings. 

I noticed and complimented others' accomplishments. 

I got along with students who were different from me. 

I was able to describe my thoughts and feelings in ways that others 
understood. 

I respected the views of others even if I disagreed with them. 

I was able to stand up for myself without putting others down. 

I knew how to disagree without starting an argument. 


0.57 
0.82 
0.81 
0.70 


0.55 
0.73 
0.57 
0.45 
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Table 6. Exploratory Factor Analysis of CC Student Survey: Factor Loadings with Oblimin Rotation 


Factor Loadings 


Item Wording 1 2 3 4 5 6 7 


Cultural and Linguistic Competence 
This school provides instructional materials that reflect my cultural background. 0.41 
Adults working at this school treat all students respectfully. 0.60 
People of different cultural backgrounds, races or ethnicities get along well at 0.63 
All students are treated the same, regardless of whether their parents are rich or 
poor. 0.75 
Boys and girls are treated equally well. 0.76 


Teacher-Student Relationship 
How many of your teachers are respectful towards you? 0.47 
If you walked into class upset, how many of your teachers would be concerned? 0.66 
If you came back to visit class three years from now, how many of your teachers 
would be excited to see you? 0.78 
When your teachers ask how you are doing, how many of them are really interested 
in your answer? 0.73 
How many of your teachers would you be excited to have again in the future? 0.71 


School Safety 
How often are people disrespectful to others at your school? 0.48 
How often do students get into physical fights at your school? 0.67 
How likely is it that someone from your school will bully you online? 0.50 
How often do you worry about violence at your school? 0.77 
If a student is bullied in school, how difficult is it for him/her to get help from an 
adult? 0.52 
At your school, how unfairly do the adults treat the students? 0.29 0.32 


Sense of Belonging 
How well do people at your school understand you as a person? 0.59 
How connected do you feel to the adults at your school? 0.38 0.23 
How much respect do students in your school show you? 0.60 
How much do you matter to others at this school? 0.64 
Overall, how much do you feel like you belong at your school? 0.41 


Learning Strategies 

When you get stuck while learning something new, how likely are you to try a 
different strategy? 0.66 
How confident are you that you can choose an effective strategy to get your 
schoolwork done well? 0.73 
Before you start on a challenging project, how often do you think about the best 0.66 
Overall, how well do your learning strategies help you learn more effectively? 0.71 
How often do you use strategies to learn more effectively? 0.81 


Engagement 
How excited are you about going to your classes? 0.80 
How often do you get so focused on activities in your classes that you lose track of 
time? 0.59 
In your classes, how eager are you to participate? 0.65 
When you are not in school, how often do you talk about ideas from your classes? 0.65 
Overall, how interested are you in your classes? 0.83 


Factor 7: Rigorous Expectations 
How often do your teachers make you explain your answers? 0.64 
When you feel like giving up on a difficult task, how likely is it that your teachers 
will make you keep trying? 0.67 
How much do your teachers encourage you to do your best? 0.72 
How often do your teachers take time to make sure you understand the material? 0.59 
Overall, how high are your teachers! expectations of you? 0.59 
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Table 7. Confirmatory Factor Analysis Model Fit Indices by Grade 


Student-Reported SEC (Spring 2017) Student-Reported CC (Spring 2017) 

CFI TLI RMSEA SRMR CFI TLI RMSEA SRMR 
Grade 4 0.995 0.995 0.030 0.062 0.985 0.983 0.042 0.068 
Grade 5 0.993 0.992 0.042 0.057 0.989 0.988 0.044 0.057 
Grade 6 0.997 0.996 0.040 0.051 0.989 0.988 0.060 0.063 
Grade 7 0.992 0.991 0.051 0.055 0.989 0.988 0.056 0.057 
Grade 8 0.993 0.992 0.048 0.066 0.991 0.991 0.045 0.070 
Grade 9 0.991 0.990 0.056 0.056 0.990 0.989 0.053 0.057 
Grade 10 0.986 0.985 0.067 0.068 0.990 0.989 0.052 0.064 
Grade 11 0.982 0.980 0.059 0.088 0.989 0.987 0.043 0.081 
Grade 12 1.000 1.000 0.000 0.079 1.000 1.000 0.000 0.082 

Discussion 


In this study, we examined the measurement properties of the SEC and CC surveys 
administered to students in the NewSchools Invent cohort of schools, using data from the Fall of 
2016 and Spring of 2017. We find that both SEC and CC scales demonstrate high internal 
consistency and that the items appear to discriminate appropriately across students’ perceptions 
of their SEC and school CC. For a few select items, most students appear to be answering them 
in the same way, suggesting that the items do not provide differentiating information. However, 
since there is no consistent pattern across grade levels, and the items fit well to expected factor 
structures, we will continue to include these items in the survey and re-examine them after we 
have additional data from subsequent years of the partnership. If, at that time, we continue to see 
evidence that the identified items do not provide differentiating information, we will explore 
ways to improve the variability in student responses without changing the underlying concept the 
item is intended to measure. 

We have identified select items that show evidence of differential item functioning across 


subgroups and between administrations; however, we do not believe immediate change to the 
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items is warranted. First, there is a lack of consistency among most items identified for DIF 


across subgroups and grade levels. We need more data and more evidence in order to draw 
conclusions on how and why these items function differently between student subgroups (Scott 
et al., 2009). Therefore, we will re-examine items when we have additional years of data, and 
flag those identified for DIF consistently across multiple terms, grade levels and years. Second, 
DIF is intended to highlight items on which students from different subgroups have differential 
responses, holding constant their overall scores on the scale or survey. However, an item flagged 
for DIF does not necessarily imply item bias. For example, differential functioning of the CC 
items may highlight important differences across subgroups in students’ perceptions or 
experiences of a particular aspect of the school culture or climate (Raju et al., 2002). To 
understand whether differential functioning suggests bias, additional evidence is required; for 
example, evidence that an item contains content that is irrelevant to the construct being measured 
or that the item assesses content relevant only to a particular cultural conception of that construct 
(Zumbo, 1999). We intend to investigate the selected items in greater detail through future 
analyses with additional years of data and through student cognitive interviews, which can help 
illuminate why students from different subgroups are responding differently on particular items. 
Exploratory factor analysis indicates potential cross-loadings of some items from the 
perseverance scale onto the self-regulation scale, suggesting there may be overlap in the 
underlying constructs being assessed. In fact, both constructs are viewed by personality 
psychologists as sub-facets of conscientious (Poropat, 2009). Further, Wolters & Hussain (2014) 
found that perseverance of efforts under the Grit Short Scale (Duckworth & Quinn, 2009) was 


consistently associated with all indicators of the self-regulated learning scale. This association 
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points to the possibility that the perseverance and self-regulation items may be conceptually 
similar. 

We also find cross-loading of two items from the perseverance scale onto the curiosity 
scale, suggesting that items from each scale are measuring the same underlying construct. This 
could be due to the fact that the identified items from the perseverance scale are quite similar to 
those in the curiosity scale. For example, an item from the perseverance scale, “I work very hard. 
I keep working when others stop to take a break,” is quite similar conceptually to an item from 
the curiosity scale, “I find satisfaction in thinking hard and for long hours.” Alternatively, it may 
be that students are responding similarly to both scales because all curiosity and perseverance 
items are grouped together in the same section on the survey. To explore the latter possibility, we 
have changed the survey for one-third of the students in the Spring 2018 test administration, in 
which the perseverance items and curiosity items have been placed into separate survey sections. 
We will then examine factor structure for those respondents for whom the survey remained the 
same, and compare it to the factor structure for those respondents for whom the two scales were 
placed in distinct sections of the survey. 

Overall, results from the EFA and CFA suggest that our survey scales are capturing 
distinct constructs. Researchers recommend making modifications only when it makes 
theoretical sense and other fit indices also do not perform as expected (Schreiber et al., 2006). 
Because of the fit of the data to the model, we recommend keeping the design of the survey as is 
and continuing to investigate the fit when additional years of data become available. We further 
advise against immediate changes to the included scales since the survey is intended to serve as a 


common measure across schools based on prioritized constructs identified by school leaders. As 
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such, additional input from these school leaders would be warranted before removing or 
modifying a survey scale. 
Conclusion 

While the individual SEC and CC scales were curated from existing surveys, this is the 
first study to examine the measurement properties of these scales when administered as cohesive 
surveys of student SEC and school CC in a unique sample of schools that are part of the 
NewSchools Invent cohort. Our preliminary results suggest that the SEC and CC surveys scales 
are suitable for practitioners to use to inform specific classroom strategies or instructional 
practices. Of course, validation is an ongoing process in which multiple sources of evidence 
should be brought to bear (Kane, 2006; 2013; Messick, 1989, 1995). In this paper, we focused 
primarily on examining how well the surveys were designed — that is, the extent to which the 
items provide consistent and new information about the underlying constructs being assessed, the 
extent to which the items are interpreted comparably across student subgroups, and the extent to 
which scales for each survey are measuring unique constructs. Based on our results, there are 
two primary aspects of survey design that require further investigation: whether particular items 
are truly interpreted differentially across subgroups (and if so, why), and the extent to which 
select SEC scales are measuring overlapping constructs. We will continue to explore both 
questions using qualitative and quantitative analyses of additional years of data. Finally, through 
an ambitious research and learning agenda with our NewSchools partnership, we will continue to 
accumulate validity evidence in order to better understand survey strengths and limitations for 


practitioner use when making data-informed decisions within and across schools. 
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Appendix A 


Table Al. Survey Items and Scales: Student-reported Social Emotional Competencies 


(reverse coded) 


If I am not naturally smart in a subject, I will never do 
well in it. 


Scale Items Response Options Source 
I like complex problems more than easy problems. Not at all like me to NAEP — General Student 
I like activities that challenge my thinking abilities. Very much like me Questionnaire 2016 Pilot 
I enjoy situations where I will have to think about 
something. Link to source (p. 15): 
Curiosity I find satisfaction in thinking hard and for long hours. https://nces.ed.gov/nationsrepor 
I enjoy thinking about new solutions to problems. tcard/subject/about/pdf/bgq/stud 
ent/2016_bq_student_g4 core_ 
p.pdf 
I like to think of my life as a puzzle that I must solve. 
Challenging myself won't make me any smarter. Not at all true to CORE & BCRC -- Farrington et 
There are some things I am not capable of learning. Completely true al. (2013) Becoming Effective 
My intelligence is something that I can't change very Learners Survey Development 
much. Project, Chicago Consortium 
Growth Mindset for School Research. 


Link to paper: 
https://consortium.uchicago.edu 
/sites/default/files/publications/ 
Noncognitive%20Report.pdf 


Perseverance 


I finish whatever I begin. 


I work very hard. I keep working when others stop to 
take a break. 


I stay interested in my goals, even if they take a long 
time (months or years) to complete. 


I am diligent. I never give up. 


Not at all like me to 
Very much like me 


BCRC — Adapted from 
Duckworth 


Link to paper: 

https://globaled.gse.harvard.edu 
/files/geii/files/validation_grit_s 
cale_duckworth jpa_m. figuer 


oa-2.pdf 


36 


‘5 


tranoforming 
education 


Link to source: 
https://www.dropbox.com/s/m5 
wo3y0iisOqtf/8- 
item%20Grit%20Scale_Child% 
20Adapted%20Version 4.pdf?d 
1=0 
Knowing what my strengths are. Very difficult to Very | Washoe — 2016 WCSD-CASEL 
Knowing ways I calm myself down. easy Long Version Items, developed 
Self-Awareness | Knowing when my feelings are making it hard for me to through IES grant 
focus. 
Link to source: 
https://www.washoeschools.net/ 
Knowing the emotions I feel. Page/10932 
I can earn an A in my classes. Not at all confident to | CORE & BCRC -- Adapted 
I can do well on all my tests, even when they are Completely confident | from Farrington et al. (2014) 
difficult. Becoming Effective Learners 
I can master the hardest topics in my classes. Survey Development Project, 
Self-Efficacy Chicago Consortium for School 
Research. 
Link to paper: 
https://consortium.uchicago.edu 
/sites/default/files/publications/ 
I can meet all the learning goals my teachers set. Noncognitive%20Report.pdf 
I came to class prepared. Almost never to CORE & BCRC -- Adapted 
I remembered and followed directions. Almost all the time from Patrick & Duckworth 
I got my work done right away instead of waiting until (2013, May) Empirical support 
: the last minute. for a tripartite taxonomy of 
Sense anon ig paid attention and resisted distractions. character in adolescents. Poster 
I worked independently with focus. presented at the 25th annual 
I remained calm even when criticized or otherwise convention of the Association 
provoked. for Psychological Science. 
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I allowed others to speak without interruption. 


I was polite to adults and peers. 


I kept my temper in check. 


Link to paper: 
https://www.researchgate.net/pu 
blication/280529814 A_triparti 
te_taxonomy_of_character 


Link to source: 
https://www.dropbox.com/s/stll 
4oeia6 1mm9k/DSIS- 


C.pdf?dl=0 


Social Awareness 


I listened carefully to other people's points of view. 


I cared about other people's feelings. 


I noticed and complimented others' accomplishments. 


I got along with students who were different from me. 


I was able to describe my thoughts and feelings in ways 
that others understood. 


I respected the views of others even if I disagreed with 
them. 


I was able to stand up for myself without putting others 
down. 


I knew how to disagree without starting an argument. 


Almost never to 
Almost all the time 


BCRC — Adapted from CASEL 
& AIR / Washoe County Items 
in 2013 


Link to source: 
https://www.washoeschools.net/ 


Page/10932 
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Table A2. Survey Items and Scales: Student-Reported Culture-Climate 


Scale 


Items 


Response Options 


Source 


Cultural and 
Linguistic 
Competence 


(reverse coded) 


This school provides instructional materials that reflect 
my cultural background, ethnicity and identity. 


Adults working at this school treat all students 
respectfully. 


People of different cultural backgrounds, races or 
ethnicities get along well at 


All students are treated the same, regardless of whether 
their parents are rich 


Boys and girls are treated equally well. 


Strongly Agree to 
Strongly Disagree 


The cultural and linguistic 
competence scale was drawn 
from the EDSCLS. It is 
based on a 4-point Likert 
Scale. For the purposes of 
our survey, we converted it 
to a 5-point Likert scale to be 
consistent with the other six 
CC scales drawn from 
Panorama. 


Link to source: 

https://safesupportivelearnin 
g.ed.gov/sites/default/files/E 
DSCLS _Questionnaires_112 


017.pdf 


How excited are you about going to your classes? 


Not at all excited to 
Extremely excited 


How often do you get so focused on activities in your 
classes that you lose track of time? 


Almost never to 
Almost always 


Not at all eager to 


strategy to get your schoolwork done well? 


Engagement In your classes, how eager are you to participate? Extremely eager 
When you are not in school, how often do you talk about | Almost never to 
ideas from your classes? Almost always 
Not at all interested to 
Overall, how interested are you in your classes? Extremely interested 
When you get stuck while learning something new, how | Not at all likely to 
Learning likely are you to try a different strategy? Extremely likely 
Strategies How confident are you that you can choose an effective | Not at all confident to 


Extremely confident 


The teacher-student 
relationship scale, school 
safety scale, sense of 
belonging scale, learning 
strategies scale, student 
engagement scale, and 
rigorous expectations scale 
were developed by Panorama 
Education. 


Link to source: 
https://www.panoramaed.co 
m/panorama-student-survey 
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Before you start on a challenging project, how often do 
you think about the best way to approach the project? 


Almost never to 
Almost always 


Overall, how well do your learning strategies help you 
learn more effectively? 


Not well at all to 
Extremely well 


How often do you use strategies to learn more 
effectively? 


Almost never to 
Almost always 


How often do your teachers make you explain your 
answers? 


Almost never to 
Almost always 


When you feel like giving up on a difficult task, how 
likely is it that your teachers will make you keep trying? 


Not at all likely to 
Extremely likely 


Do not encourage me 


(reverse coded) 


Rizcrous How much do your teachers encourage you to do your at all to Encourage me 
Expectations 
best? a tremendous amount 
How often do your teachers take time to make sure you _| Almost never to 
understand the material? Almost always 
Not high at all to 
Overall, how high are your teachers' expectations of you? | Extremely high 
How often are people disrespectful to others at your Almost never to 
school? Almost always 
How often do students get into physical fights at your Almost never to 
school? Almost always 
School Safety How likely is it that someone from your school will bully | Not at all likely to 
you online? Extremely likely 


How often do you worry about violence at your school? 


Almost never to 
Almost always 


If a student is bullied in school, how difficult is it for 
him/her to get help from an adult? 


Not at all difficult to 
Extremely difficult 


At your school, how unfairly do the adults treat the 
students? 


Not at all unfairly to 
Extremely unfairly 


Sense of 
Belonging 


How well do people at your school understand you as a 
person? 


Do not understand at 
all to Completely 
Understand 
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How connected do you feel to the adults at your school? 


Not at all connected to 
Extremely connected 


How much respect do students in your school show you? 


No respect at all to A 
tremendous amount of 
respect 


How much do you matter to others at this school? 


Do not matter at all to 
Matter a tremendous 
amount 


Overall, how much do you feel like you belong at your 
school? 


Do not belong at all to 
Completely belong 


Teacher-Student 
Relationship 


How many of your teachers are respectful towards you? 


If you walked into class upset, how many of your 
teachers would be concerned? 


If you came back to visit class three years from now, 
how many of your teachers 


When your teachers ask how you are doing, how many of 
them are really interested in your answer? 


How many of your teachers would you be excited to 
have again in the future? 


None of my teachers 
to All of my teachers 
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Appendix B 


Table B1. Spring 2017 Student SEC Summary Descriptive Statistics by Subgroup 


Growth Self- Self- Self- Social 
% Curiosity | Mindset Perseverance Awareness Efficacy Regulation Awareness 
3.21 3.79 3.58 3.65 3.45 3.90 3.77 
Overall n=3,028 | (n=3,015) — (n=3,012) (n=3,011) (n=3,012) — (n=3,018) —_ (n=3,027) (n=3,016) 
3.15 3.84 3.57 3.59 3.35 3.98 3.84 
Female 49% (n=1,483)  (n=1,482) (n=1,482) (n=1,482) (n=1,486) = (n=1,491) (n=1,486) 
3.27 3.74 3.59 3.71 3.53 3.83 3.71 
Male 51% (n=1,531) — (n=1,529) (n=1,528) (n=1,529) — (n=1,531) —_ (n=1,535) (n=1,529) 
3.14 3.72 3.55 3.57 3.28 3.88 315 
Latino 53% (n=1,596) (n=1,596) (n=1,596) (n=1,595) (n=1,598) = (n=1,599) (n=1,598) 
3.32 3.68 3.60 3.73 3.55 3.77 3.66 
Black 18% (n=541) (n=541) (n=540) (n=541) (n=542) (n=547) (n=541) 
3.46 4.13 3.75 3.83 3.82 4.11 4.05 
Asian 3% (n=82) (n=82) (n=82) (n=82) (n=82) (n=82) (n=82) 
3.14 3.81 3.59 3.79 3.63 4.00 3.81 
Two or More Races 1% (n=37) (n=37) (n=37) (n=37) (n=38) (n=38) (n=37) 
3.30 4.03 3.65 3.80 3.77 4.07 3.93 
White 21% (n=633) (n=631) (n=631) (n=631) (n=633) (n=634) (n=632) 
3.21 3.80 3.53 3.91 3.31 4.00 3.86 
Other 0.53% (n=16) (n=16) (n=16) (n=16) (n=16) (n=16) (n=16) 
3.02 3.73 3.37 3.32 3.16 3.55 3.63 
MENA 3% (n=87) (n=86) (n=86) (n=87) (n=86) (n=88) (n=87) 
3.16 3.46 3.46 3.53 3.21 3.76 3.64 
ELL 18% (n=520) (n=520) (n=520) (n=520) (n=521) (n=522) (n=521) 
3.22 3.86 3.60 3.67 3.48 3.94 3.80 
Non-ELL 82% (n=2,392) — (n=2,389) (n=2,389) (n=2,389) — (n=2,394) —_ (n=2,401) (n=2,394) 
3.19 3.72 3.56 3.61 3.35 3.88 3.15 
FRPL 64% (n=1,780) (n=1,777) (n=1,776) (n=1,778)  (n=1,782) = (n=1,788) (n=1,780) 
3.30 3.97 3.64 3.76 3.70 4.01 3.88 
Non-FRPL 36% (n=1,004) — (n=1,003) (n=1,003) (n=1,003) — (n=1,004) (1,007) (1,005) 
3.23 3.40 3.45 3.53 3.30 3.66 3.53 
SPED 8% (n=237) (n=237) (n=237) (n=236) (n=239) (n=239) (n=238) 
3.21 3.82 3.59 3.66 3.46 3.92 3.79 
Non-SPED 92% (n=2,775) _ (n=2,772) (n=2,771) (n=2,773) (2,776) (n=2,785) (n=2,775) 
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Table B2. Spring 2017 Student CC Summary Descriptive Statistics by Subgroup 


Cultural & Teacher- 
Linguistic Learning Rigorous School Sense of Student 
% Competence Engagement Strategies Expectations Safety Belonging Relationship 
3.67 2.90 3.51 3.89 3.80 3.26 3.52 
Overall n=3,092 (n=3,086) (n=3,074) (n=3,078) (n=3,080) (n=3,083) _ (n=3,083) (n=3,085) 
3.69 2.89 3.53 3.91 3.82 3.24 3.51 
Female 50% (n=1,530) (n=1,525) (n=1,528) (n=1,529) (n=1,529)  (n=1,529) (n=1,528) 
3.66 2.91 3.50 3.87 3.80 3.29 3.54 
Male 50% (n=1,555) (n=1,548) (n=1,549) (n=1,550) (n=1,553) — (n=1,553) (n=1,556) 
3.70 2.83 3.46 3.85 3.85 3.21 3.43 
Latino 52% (n=1,604) (n=1,600) (n=1,603) (n=1,605) (n=1,605)  (n=1,605) (n=1,604) 
3.49 3.00 3.51 3.87 3.60 3.24 3.53 
Black 18% (n=555) (n=552) (n=552) (n=553) (n=554) (n=554) (n=555) 
3.88 3.18 3.77 4.15 3.91 3.54 4.02 
Asian 3% (n=88) (n=87) (n=87) (n=87) (n=87) (n=87) (n=87) 
3.51 3.03 3.70 4.06 3.81 3.48 3.74 
Two or More Races 1% (n=39) (n=38) (n=38) (n=38) (n=38) (n=38) (n=39) 
3.79 2.98 3.62 3.99 3.89 3.39 3.75 
White 22% (n=672) (n=669) (n=670) (n=669) (n=671) (n=671) (n=672) 
3.58 2.64 3.43 3.64 3.94 2.66 3.21 
Other 0.49% (n=15) (n=15) (n=15) (n=15) (n=15) (n=15) (n=15) 
3.25 2.53 3.42 3.44 3.56 3.11 3.03 
MENA 3% (n=91) (n=91) (n=91) (n=91) (n=91) (n=91) (n=9 1) 
3.66 2.97 3.44 3.80 3.69 3.32 3.56 
ELL 18% (n=539) (n=537) (n=539) (n=539) (n=539) (n=539) (n=539) 
3.67 2.87 3.52 3.90 3.83 3.24 3.49 
Non-ELL 82% (n=2,334) (n=2,330) (n=2,329) (n=2,330) 2,332 (n=2,332) (n=2,334) 
3.65 2.91 3.51 3.89 3.81 3.26 3.49 
FRPL 64% (n=1,814) (n=1,812) (n=1,814) (n=1,814) (n=1,815) (n=1,815) (n=1,815) 
3.72 2.95 3.60 3.98 3.80 3.32 3.65 
Non-FRPL 36% (n=1,035) (n=1,024) (n=1,026) (n=1,028) (n=1,030) _(n=1,030) (n=1,032) 
3.53 2.95 3.37 3.70 3.61 3.23 3.62 
SPED 9% (n=264) (n=262) (n=262) (n=261) (n=263) (n=263) (n=263) 
3.69 2.89 3.53 3.90 3.82 3.26 3.51 
Non-SPED 91% (n=2,819) (n=2,810) (n=2,814) (n=2,817) (n=2,818) (n=2,818) (n=2,820) 
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Table C1. Cronbach's Alpha by Grade: Student-Reported SEC Constructs (Spring 2017) 
Constructs 
ee Growth Self- Self- Self- Social 
Curiosity Mindset Perseverance Awareness Efficacy Regulation Awareness 
Grade 4 0.84 0.70 0.74 0.73 0.85 0.84 0.85 
Grade 5 0.86 0.78 0.79 0.71 0.87 0.87 0.87 
Grade 6 0.89 0.80 0.84 0.80 0.92 0.92 0.91 
Grade 7 0.89 0.79 0.83 0.76 0.90 0.89 0.87 
Grade 8 0.89 0.79 0.84 0.76 0.89 0.89 0.88 
Grade 9 0.89 0.79 0.84 0.77 0.91 0.88 0.89 
Grade 10 0.87 0.82 0.83 0.72 0.90 0.89 0.88 
Grade 11 0.87 0.82 0.79 0.71 0.88 0.81 0.88 
Grade 12 0.89 0.90 0.87 0.83 0.91 0.90 0.92 


Table C2. Cronbach's Alpha by Grade: Student-Reported CC Constructs (Spring 2017) 


Constructs 
Grade Cultural & Teacher- 
Linguistic Learning Rigorous School Sense of Student 
Competence Engagement Strategies Expectations Safety Belonging Relationship 

Grade 4 0.66 0.77 0.73 0.74 0.71 0.81 0.83 
Grade 5 0.77 0.84 0.79 0.78 0.71 0.79 0.84 
Grade 6 0.80 0.83 0.86 0.85 0.76 0.84 0.89 
Grade 7 0.81 0.84 0.83 0.84 0.74 0.84 0.88 
Grade 8 0.79 0.85 0.86 0.83 0.68 0.83 0.86 
Grade 9 0.82 0.85 0.87 0.84 0.72 0.81 0.88 
Grade 10 0.84 0.86 0.88 0.83 0.72 0.81 0.85 
Grade 11 0.80 0.81 0.84 0.84 0.70 0.76 0.84 
Grade 12 0.82 0.89 0.90 0.90 0.79 0.86 0.91 
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